chore(streaming): re-sync #784 with main after #810, #816, #818 and #822 - #824

…atch (#816)

Python's $ also matches just before a final "\n", so
re.match(r"^[a-z]+$", "orders\n") succeeds. #788 and #797 fixed the known
validators with \Z or fullmatch, but nothing kept the class from coming
back: no ruff, pylint, bandit, dlint or Semgrep CE rule covers it
(astral-sh/ruff#15017 proposes one).

tests/security/test_no_dollar_anchored_validators.py walks the AST of
fluid_build/ and follows the pattern of every .match / .search call back
to where it is written: a literal or constant, a dict or tuple, an import,
a helper's return value, a property, a re.Pattern parameter, or a pattern
compiled at run time. It reads each pattern with re's own parser, so
alternatives, groups, escapes and VERBOSE count the way re reads them.
Calls whose $ is deliberate are listed in _ACCEPTED with the reason. An
entry that no longer matches a call fails as well, so the list only
shrinks.

On main before #797 it fails on 44 calls, including the shapes the sweep
missed: a pattern held in a dict, schema patterns compiled at run time,
and a pattern the caller passes in. It also caught the $ + .match sites
merged since (#793, #801, #799). Each was read and is listed with its
reason.

A clean merge of origin/main (e7f6bca) onto #821's head, 369d088. It
brings in #810 (the welcome-scan walk is bounded and ends with its scan)
and #816 (a guard against whole-string validators that end in `$` under
.match).

Neither touches a feature file. The merge adds exactly main's
85174c0..e7f6bca patch (same patch-id). #816's guard passes on the
feature code: 73 passed.

… operator paired it with (#818)

* security(iceberg-sink): pair a catalogAuth variable with every host it reaches

On the Kafka Connect and embedded Debezium Iceberg sinks, sink.catalogAuth
names an environment variable and the same contract names the catalog
(location.uri) and the OAuth2 token endpoint. Iceberg resolves the emitted
env:<NAME> on the Connect worker, or on embedded Debezium from the runner's
whole environment, so a contract could send any variable there to a host it
picked. A probe with a catalog at https://catalog.attacker.example returned
rc 0 on both engines and both auth types.

The sink preflight, and fluid validate as a warning, now require for the
catalog and an OAuth2 token endpoint:
- https, loopback included, as on the managed Flink lane;
- NAME@host in FLUID_ACQUISITION_SECRET_REF_ALLOWLIST (loopback exempt);
- a hostname listed in FLUID_ACQUISITION_DEPLOYMENT_HOST_ALLOWLIST, since
  the worker resolves it itself; IP literals keep the #758 rules.
The check runs before any Connect REST call, application.properties write
or read of the variable. Location text the derived config carries may not
start with env: or hold ${ or a control character, and client.region must
be one host label. A {{ env.* }} template is checked at run, where the
dispatcher renders it.

Ported from the managed Flink lane's policy (#809, streaming/policy.py).
Host-scoped credentials follow Terraform's per-host TF_TOKEN_<host>; an
operator list of readable variables follows Kafka's KIP-887
EnvVarConfigProvider allowlist.pattern. The host rule and the https scope
were chosen by the founder on 2026-11-04.

* docs(changelog): an Iceberg sink's catalog credential needs an operator pairing

* fix(iceberg-sink): the destination refusal gives one export line to paste

Both operator lists now come in a single export, as on the managed Flink lane, and the sentence about where Iceberg resolves the variable no longer runs two relative clauses together.

* refactor(iceberg-sink): split the destination check so each helper decides one thing

No behaviour change. _check_catalog_auth keeps its size: one call checks the hosts, then reads the embedded Debezium variable. The destinations, the hard refusals, the host-list rule and the message are separate helpers (ripwire flagged complexity 34 and a verbosity regression on _check_catalog_auth).

* docs(iceberg-sink): name no lane main lacks; say which refusals validate warns about

The https refusal no longer cites the managed Flink lane, which is not on main. The module docstring says https and link-local are errors in fluid validate too, the env: note is scoped to the REST catalog with the rule kept for every kind as hardening, and the CHANGELOG adds the location-text upgrade note and says a bare NAME or a parent domain grants more than NAME@host.

* security(iceberg-sink): refuse a credential host the refusal cannot repeat safely

The refusal's export line repeated the contract's host inside double
quotes, so a catalog uri such as https://x"$(id)".example/ turned the
operator's paste into a command, and a comma in the host added a bare
entry that lets another variable go to any host. A credential destination
must now name a plain DNS name or IP address, with no zone id and no
user:password@ part (Python and the JVM split such an authority
differently). The binding.location text check runs only for the two
engines whose runners derive the sink config; any other engine raised
KeyError there. Found by the scoped codebase audit.

* fix(iceberg-sink): the judge's lows: no rendered region in a message, path templates checked now, no read for a non-REST kind

A run-time region refusal no longer quotes the value, which a {{ env.* }} template may have rendered from a secret. fluid validate defers a templated URL only when the template sits in the scheme or authority, so http://169.254.169.254/{{ env.X }} is refused at validate. A catalogAuth on a catalog not reached over REST is already refused, and the embedded preflight now reads no variable for it.

* fix(iceberg-sink): check the fields the sink reads, and a written-out scheme

From the scoped defect scan:
- sink.partitionBy, which the runner's SinkSpec reads over the binding's,
  is now the partition text checked when set; it reached both emitters
  unchecked.
- Only the namespace and table fields the sink reads are checked (the first
  set, as iceberg_namespace reads them), so an unused field no longer
  fails a build, and a derived warehouse names the location fields it came
  from.
- A template in a URL's host still leaves the host to the run, but an
  http:// scheme written out is refused by fluid validate.
- An IP literal is parsed as written: 10.0.0.1%25.example is a name.
- The refusal says its export line keeps the entries set where it ran.
- test_catalog_auth_sink extends the host list after kafka_connect_mock,
  which may set it for its own server (the pinned-address branch does).

* test(iceberg-sink): pin the cases the test audit found open; skip the live OAuth2 trio until TLS

The destination tests now clear an ambient host list, assert that the
export line keeps the operator's host entries, and cover a URL with no
readable host, a token endpoint judged apart from a loopback catalog
(remote, link-local, unspecified), the validate warning on both engines
and both types, and a Glue region.

tests/integration/test_lakekeeper_oauth_live.py serves Lakekeeper and
Keycloak over http, which a catalogAuth credential no longer reaches. The
three tests that derive a sink config are skipped until the stack runs
TLS; the founder chose that on 2026-11-04 over moving the stack in this
PR. The fourth test talks to Lakekeeper directly and still runs.

* test(iceberg-sink): mark the synthetic secrets detect-secrets flags

AWS's documented example key and a u:p4ss@ userinfo are test values; each pragma sits on the line that holds the value, where black leaves it.

* test(iceberg-sink): reuse the catalogAuth sink test contracts and fake server

The destination tests cloned _kc_contract, _dbz_contract and the fake Debezium server from test_catalog_auth_sink.py. They now call those, which take an optional location, and share one installer for the fake server (ripwire flagged the clones).

* fix(iceberg-sink): read sink.partitionBy exactly as SinkSpec does

A mapping-shaped partitionBy reaches the config as its keys (SinkSpec.from_dict lists it), so the check lists it the same way rather than falling back to the binding's. A value SinkSpec cannot list stops the run anyway. From the judge's re-review.

* test(iceberg-sink): point the skipped live trio at its follow-up card

Brings in #818 (a catalogAuth variable reaches only hosts the operator
paired it with). Base e7f6bca.

Resolution, one conflict:
- fluid_build/build_runners/_secret_policy.py: #809 and #818 each added the
  same `variable_allowed` function, with identical code. Only the docstring
  differed, naming each side's caller. Kept one function whose docstring
  names both callers: the managed Flink lane's catalog token
  (streaming/policy.py) and the Iceberg sinks' catalog token
  (kafka_connect/iceberg_sink_validation.py).

The file differs from main only in that docstring. Every other #818 file
merged without conflict and is not a feature file.

…ay why Databricks M2M is refused (#822)

tests/test_litellm_endpoint_names.py read only names ending in _API_BASE,
so DASHSCOPE_, QWENCLOUD_ and QWEN_AI_PLATFORM_API_BASE_{IMAGE,RERANK,VIDEO}
(read by litellm's dashscope provider code with get_secret_str; the VIDEO
three are new in 1.108.0, the rest date from 1.83.14 to 1.105.0) were never
in LITELLM_ENDPOINT_NAMES. The check now reads a kind after _API_BASE and
the nine are listed; with the wider check alone, the drift test fails on
litellm 1.108.0. Nothing reached them: FLUID calls only litellm.completion.

The Databricks model-serving group also lists DATABRICKS_CLIENT_SECRET:
litellm's OAuth M2M path (llms/databricks/common_utils.py) takes the
workspace from DATABRICKS_API_BASE and posts the secret there. The
name-shape rule already refused a project DATABRICKS_API_BASE while the
operator holds that secret; a new test pins it, and the group names the
reason. Both found by an Opus review of #820.

Brings in #822 (litellm's per-kind endpoint routes are listed, and the
Databricks M2M refusal says why). A clean merge with base 6f68f64. #822
touches CHANGELOG.md, credentials/litellm_endpoints.py,
credentials/protected_env.py and the endpoint-names test, none of them
feature files. The merge adds exactly #822's patch. The feature's
FLUID_FLINK_*_ALLOWLIST variables stay dotenv-protected.